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ABSTRACT 



A series of pilot studies was conducted to evaluate the 
instructional effectiveness of the Harcourt Reading Program. One of the 
studies was conducted in the fall of 1998-99 by Indiana University, and data 
were analyzed by them also. Harcourt School Publishers sought out teachers to 
participate. Teachers from New Jersey, New York, Ohio, and Pennsylvania 
volunteered, 14 at grade 1 and 12 at grade 4. The following research 
questions guided the design, procedures, and data analysis: Is Harcourt 
Reading instructionally effective? and Does Harcourt Reading have a positive 
impact on students' attitudes and interests? A quasi-experimental 
pretest-posttest design was used. Following the pretests, the teachers taught 
selected themes using the resources and procedures contained in Harcourt 
Reading. Table 1 summarizes the skills taught in the themes piloted by the 
teachers in the study and indicates that the pilot study included a broad 
sample of reading skills from the strands of decoding, vocabulary, 
comprehension, literary appreciation, and study skills. Tables 2, 3, and 4 
show that students realized gains in reading scores after using the program. 
Findings suggest that Harcourt Reading is instructionally effective, and this 
appears to hold up across grade levels. Contains 4 tables of data. (NKA) 



***************************************************************************** 

♦ Reproductions supplied by EDRS are the best that can be made 

♦ from the original document . 



CS 013 592 






r~i 

TT 

Q 

W 



HARCOURT READING 
Technical Report #2 

A Study of the Instructional Effectiveness 
of the HARCOURT READING Program 



Prepared by 

The Center for Innovation in Assessment 
Indiana University 

and 

Department of Test Services 
Harcourt Brace School Publishers 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
y CENTER (ERIC) 

H^This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



HARCOURT® 
BRACE 



May 1999 



permission to reproduce and 
disseminate this material has 

BEEN GRANTED BY 

E>. 



TO THE EDUCATIONAL RESOURCES 
information center (ERIC) 



o 

ERIC 



2 



BEST COPY AVAILABLE 



A Study of the Instructional Effectiveness 
of the HARCOURT READING Program 



This report describes one of a series of pilot studies that were conducted to evaluate the 
instructional effectiveness of the HARCOURT READING program. 

Background Information 

Harcourt School Publishers asked the Center for Innovation in Assessment at Indiana 
University to conduct a series of independent pilot studies to evaluate the effectiveness of 
HARCOURT READING. Harcourt School Publishers sought out volunteer teachers to 
participate in the study; the Department of Test Services assisted in designing, securing, 
and scoring the data collection instruments; and Indiana University conducted the study 
and analyzed the data that were collected. The study described in this report was 
conducted in the fall of the 1998-99 school year. 

Research Questions 

The following research questions guided the design, procedures, and data analysis of the 
pilot study: 

Is HARCOURT READING instructionally effective? Do selected themes 
significantly increase students ’ understanding of key reading skills, concepts, and 
strategies as measured by a program Skills Test? As measured by a standardized 
achievement test? 

Does HARCOURT READING have a positive impact on students ’ attitudes and 
interests? Do selected themes significantly increase students ’ attitudes toward 
reading as measured by an informal survey? 

Design and Procedures of the Study 

Because of cost and time considerations, the study was limited to two grades - grades 1 , 
and 4. It was believed that these two grades would give a good cross-sectional look at the 
primary and upper elementary levels of the program. 

Teachers who volunteered to participate in the study had not used the program previously. 
The teachers were allowed to select one theme to pilot over a four- to six-week period. 
They also agreed to administer data collection instruments before beginning instruction 
and again after completing instruction. 
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A total of 26 teachers volunteered to participate in the study: 14 at grade 1, and 12 at 
grade 4. The participating teachers came from four states: New Jersey, New York, Ohio, 
and Pennsylvania. 

A quasi-experimental pretest-posttest design was used. Before instruction began, 
students were administered the following pretests: (1) the appropriate HARCOURT 
READING Skills Test; (2) the reading section of the Stanford 9 Achievement Test, Ninth 
Edition (Abbreviated Battery); and (3) and the Elementary Reading Attitude Survey, an 
informal instrument designed to measure attitudes toward reading. 

Following the pretests, the teachers taught the selected themes using the resources and 
procedures contained in HARCOURT READING. Teachers also received training from 
experienced consultants in methods of implementing and using the program. 

Instructional time varied from four to six weeks depending on the themes taught. In most 
cases, HARCOURT READING replaced the existing reading program. 

Upon completion of the theme, students were administered the same three instruments as 
posttests. All data collection instruments were returned to Indiana University where they 
were processed. The Stanford 9 Achievement Tests were scored at the Harcourt Brace 
Educational Measurement Scoring Center. The HARCOURT READING Skills Tests 
and the Elementary Reading Attitude Surveys were scored at Indiana University, and all 
of the data were analyzed at Indiana University. 

Table 1 summarizes the skills taught in the themes that were piloted by teachers in the 
study. As Table 1 indicates, the pilot study included a broad sample of reading skills 
from the strands of decoding, vocabulary, comprehension, literary appreciation, and study 
skills. 
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TABLE 1 

Content of the HARCOURT READING’ Themes 
Taught by the Pilot Teachers 



Theme 


Grade 1 


Grade 4 


1 


Vocabulary 


Vocabulary 




Key Words 


Key Words 




Decoding 


Comprehension 




Initial consonants 


Make Predictions 
Draw Conclusions 


2 


Vocabulary 


Vocabulary 




Key Words 


Key Words 




Decoding 


Decoding 




Short vowels; final consonants 


Structural analysis: prefixes, suffixes, 




Structural analysis; inflected words 


roots 

Literary Appreciation 
Narrative elements 


4 


Vocabulary 


Vocabulary 




Key Words 


Key Words 




Decoding 


Context clues 




Short vowels/long vowels 


Comprehension 




Structural analysis: inflected forms 


Cause and effect 




Comprehension 


Main idea/details 




Make predictions 
Study Skills 
Alphabetical order 


Sequence 



Findings 

Table 2 summarizes the pre- and posttest means and standard deviations for the Skills 
Test, the Stanford 9, and the Elementary Reading Attitude Survey for both grades. The 
means and standard deviations for the Skills Tests and the Elementary Reading Attitude 
Survey are presented in percentages; those for the Stanford 9 are in raw scores. 

TABLE 2 

Means and Standard Deviations of 
Pretests and Posttests 



Grade 


Skills 

Test 


Stanf 
Total F 


ord 9 
leading 


Attitude 

Survey 


Grade 1 (N=132) 


Mean 


SD 


Mean 


SD 


Mean 


SD 


Pretest 


64.'9 


21.1 


84.2 


16.9 


83.3 


10.0 


Posttest 


85.0 


16.1 


95.8 


13.4 


84.1 


9.4 


Grade 4 (N=126) 














Pretest 


69.9 


16.1 


35.7 


5.3 


77.1 


9.9 


Posttest 


85.3 


10.7 


38.1 


5.8 


77.3 


12.1 
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At grade 1, the mean Skills Test score increased from 64.9% before instruction to 85.0% 
after instruction. This increase was highly significant (1=15.021 ; p<.0001). Likewise, 
the raw score means for Total Reading on the Stanford 9 increased from 84.2 to 95.8. 

This gain was also significant (1=13.485; p<.0001). The raw score means from the 
Stanford 9 were converted to scale scores, percentiles, and grade equivalents in order to 
interpret the increase in normative scores. The mean scale score increased from 492 to 
521; the percentile scores for the means increased from 69 to 84; and the grade equivalent 
scores for the means increased from 1 .3 to 1 .7. The mean score on the Attitude Survey 
increased from 83.3% to 84. 1% percent. Although positive, this increase was not 
significant (t=.898; p<.371). 

At grade 4, the mean Skills Test score increased from 69.9% to 85.3%, a statistically 
significant gain (1=12.921; p<.0001). The Total Reading raw score mean on the 
Stanford 9 increased from 35.7 to 38.1. This increase was also significant (1=5.189; 
p<.0001). When interpreted in terms of normative scores, the scale score went from 627 
to 636, the percentile score from 76 to 80, and the grade equivalent score from 4.3 to 4.7. 
The mean score for the Attitude Survey increased slightly from 77.1% to 77.3%. This 
increase was not significant (t=.327; p<. 744). 

Because both groups showed a significant gain in the Stanford 9 Total Reading score, an 
additional analysis was conducted to see which subtests may have contributed to those 
gains. Table 3 shows the pre- and posttest subtest means and standard deviations for the 
Stanford 9 for grade 1 . 



TABLE 3 

Means and Standard Deviations of the 
Stanford 9 Subtests for Grade 1 



Test 


Word 

Study Skills 


Word 

Reading 


Reading 

Comprehension 




Mean 


SD 


Mean 


SD 


Mean 


SD 


Pretest 


33.5 


5.6 


32.1 


6.3 


19.6 


7.6 


Posttest 


36.0 


4.8 


35.5 


4.6 


24.7 


5.6 



As Table 3 shows, mean scores increased for all three subtests. The increases for Word 
Study Skills (t=8.276; p<0001), for Word Reading (1=10.208; p< 0001), and for 
Reading Comprehension (t= 12.68; p<.0001) were all highly significant. 

A similar analysis was conducted for grade 4. At grade 4, the Stanford 9 yields two 
subtest scores - Vocabulary and Reading Comprehension. Table 4 presents the pre- and 
posttest means and standard deviations for these subtests. 
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TABLE 4 

Means and Standard Deviations of the 
Stanford 9 Subtests for Grade 4 



Test 


Vocabulary 


Reading 

Comprebension 




Mean 


SD 


Mean 


SD 


Pretest 


15.5 


2.0 


20.2 


4.1 


Posttest 


16.4 


2.1 


21.7 


4.4 



As in grade 1 , mean scores increased on each of the subtests. The increase in Vocabulary 

((=4.476; p<.0001) and the increase in Reading Comprehension (1=3.808; p<.0001) were 

both highly significant. 

Conclusions 

The following conclusions can be made based on the findings: 

• HARCOURT READING is instructionally effective. Students using HARCOURT 
READING increase their understanding of key reading skills, concepts, and 
strategies. 

• The instructional effectiveness of HARCOURT READING appears to hold up across 
grade levels. In this study, students in grades 1 and 4 made significant gains in 
reading achievement after completing one theme. 

• The instructional effectiveness of HARCOURT READING also appears to 
demonstrate itself on both criterion-referenced measures related to the program as 
well as standardized achievement tests that are independent of the program. 

• At grade 1 , the average increase on the Stanford 9 was approximately 1 5 percentile 
points after completing instruction in only one theme, with significant gains made in 
Word Study Skills, Word Reading, and Reading Comprehension. 

• At grade 4, the average increase on the Stanford 9 was about 4 percentile points after 
completing instruction in only one theme, with significant gains made in Vocabulary 
and Reading Comprehension. 

• There were also increases in student attitudes toward reading at both grades 1 and 4, 
but the increases were not statistically significant. It is likely that attitudes and 
interests toward reading are formed over a longer period of time, and not as sensitive 
to short-term changes in instruction. 
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